In this section, we further clarify why a naive application of data augmentation with certain RL algorithms is theoretically unsound. The correct estimate of the policy gradient objective used in PPO is the one in equation(1) (or equivalently, equation(8)) which does not use the augmented observations at all since we are estimating advantages for the actual observations,A(s,a). The probability distribution used to sample advantages isπold(a|s)(rather thanπold(a|f(s))since we can only interact with the environment via the true observations and not the augmented ones (because the reward and transition functions are not defined for augmented observations). Hence, the correct importance sampling estimate usesπ(a|s)/πold(a|s). Usingπ(a|f(s))/πold(a|f(s))instead would be incorrect for the reasons mentioned above. In contrast, DrAC does not change the policy gradient objective at all which remains the one in equation(1)andinsteadusestheaugmented observationsintheadditional regularizationlosses,as showninequations (3), (4),and (5). Note that this cycle-consistency implies that two trajectories are accurately aligned in the hidden space.

The success of deep (reinforcement) learning systems crucially depends on the correct choice of hyperparameters which are notoriously sensitive and expensive to evaluate. Training these systems typically requires running iterative processes over multiple epochs or episodes. Traditional approaches only consider final performances of a hyperparameter although intermediate information from the learning curve is readily available. In this paper, we present a Bayesian optimization approach which exploits the iterative structure of learning algorithms for efficient hyperparameter tuning. First, we transform each training curve into a numeric score. Second, we selectively augment the data using the auxiliary information from the curve. This augmentation step enables modeling efficiency while preventing the ill-conditioned issue of Gaussian process covariance matrix happened when adding the whole curve. We demonstrate the efficiency of our algorithm by tuning hyperparameters for the training of deep reinforcement learning agents and convolutional neural networks. Our algorithm outperforms all existing baselines in identifying optimal hyperparameters in minimal time.